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1. Introduction 

Let f2 be a convex domain in W 1 containing the origin in its interior. We mostly assume that f2 
has smooth boundary and that the Gaussian curvature of the boundary vanishes nowhere. Let 

Mn(t) =card(^nZ d ), 

the number of integer lattice point inside the dilated domain ttt. It is well known that Afn(t) is 
asymptotic to t d vo\(fl) as t — > oo. We denote by 

the relative error, or the discrepancy function. It is conjectured that in dimensions d > 5 the relative 
error is 0(t~ 2 ) as t — > oo. This conjecture is known to be true in the case of a ball centered at 
the origin, and for ellipsoids in various degrees of generality (see Landau [20], Walfisz [31], [32], 
Bentkus and Gotze [2]). The error can be even smaller. For example, Jarmk [14] established the 
bound 0(t~ d / 2+e ) for the relative error, with any e > 0, for almost all ellipsoids with axes parallel 
to the coordinate axes. For general convex domains with non-vanishing curvature on the boundary, 
W. Miiller [22] proved that A Q (t) = 0(t- 2+x ^ +E ), where X(d) = (d + 4)/(d 2 + d + 2), if d > 5, 
A(4) = 6/17 and A(3) = 20/43, improving on earlier results by Kratzel and Nowak [19]. For planar 
domains, Huxley [11] obtained this estimate with A(2) = 46/73, which implies the relative error 

O (£-100/73( logt )315/146)_ 

In this paper we study the mean square discrepancy of the lattice rest, the square function 

/ 1 r 2R \ 1/2 

(1-2) Ga{ - R) = \Rj R I^WI*) 

and related expressions. For the ball Bd in M. d , centered at the origin, bounds and various asymptotics 
for mean square discrepancies have been obtained by Walfisz [32] for d > 4, Jarnik [16] for d = 3 
and Katai [17] for d = 2. 

In the more general situation where the boundary of f2 is smooth and is assumed to have 
everywhere non-vanishing Gaussian curvature, Nowak [25] proved that Gn(R) — 0(i?~ 3 ^ 2 ) for planar 

Research supported in part by NSF grants. 

Typeset by 



1 



domains. This estimate is sharp by the results of Bleher [3] who investigated the limit of R 3 ^ 2 Qq(R) 
as R — -> oo. The higher dimensional case was considered by W. Miiller [21], who proved a nearly 
sharp estimate for d > 4, namely that Gn{R) < C E R~ 2+£ for any e > 0. The case d = 3 was left 
open. 

The main purpose of this note is to show that the known endpoint bounds for the mean square 
discrepancy in the case of the ball remain valid in the general case, provided that d > 4. Moreover, 
we prove a nearly sharp estimate in dimension d = 3, where we are off by a factor of \/log R. 

Theorem 1.1. Let il be a convex domain in R d containing the origin in its interior, and assume 
that f2 has smooth boundary with everywhere non-vanishing Gaussian curvature. Then there exists 
a constant C(f2), such that for all R> 2, 



(1.3) Gn(R) <C(fl)< 



R- 2 ifd>4 
R~ 2 logR ifd = 3 
R~ 3 / 2 ifd = 2 



As we noted above, the sharp estimate 0{R~ 3 / 2 ) in the plane was already known for more 
general planar domains with the non-vanishing curvature assumption. In fact, it turns out that 
this estimate holds even if we replace the mean square discrepancy over [R, 2R] by the mean square 
discrepancy over substantially smaller intervals [R,R+h]. A closely related result due to Huxley 

d , i \ 1/2 

[10] says that (J* +1 \A Q (t)\ 2 dt) < C n R- 3 / 2 (log R) 1 ' 2 . 



Theorem 1.2.1. Let Q, be a convex domain in M 2 containing the origin in its interior, and assume 
that O has C°° boundary with non-vanishing curvature. Then there is a constant C — C(Cl) so that 
for all R>2, 



(1.4) \A n (t)\ 2 dty /2 < C(n)R- 3/2 if \ogR<h<R. 

As an immediate consequence of Theorem 1.2.1, the mean square discrepancy over [R, R + h] is 
dominated by C(f2)i?~ 3 / 2 (log R) x l 2 h~ x l 2 if < h < \ogR. In particular, the aforementioned result 
of Huxley follows if we set h = 1. 

We now consider more general domains in the plane. We say that a convex domain is of type 
at most m if its boundary has order of contact at most m with every tangent line. Thus if m = 2 
we recover the case of everywhere non-vanishing curvature considered above. It is known that the 
analogue of Theorem 1.2.1 may fail if the order is greater than 2 (c/. [26], [5], and [23]). However 
for almost all rotations the estimate remains true for the rotated domain. More precisely we have 
the following result. 

Theorem 1.2.2. Let Q be a convex domain in R 2 containing the origin in its interior, and assume 
that the boundary is smooth and of finite type at most m, in the sense that the order of contact 
of dfl with every tangent line is at most m. For A G SO(2), denote by AO, the rotated domain 
{Ax : x € f2}. Then for almost all rotations A, the inequality (1-4) holds for AQ, with the constant 
Caq. depending on A. More precisely, the following hold. 



(i) The maximal function 



(1.5) Co(A) = supi? 3 / 2 sup (- / \A An (t)\ 2 dt) 

R>2 log R<h<R v ^ JR ' 

2 m - 2 

belongs to the weak type space L m - 2 '°°(SO(2)). 

(ii) Let r be the set of all points P <G dfl where the curvature vanishes, and for P e T assume 
that the curvature vanishes of order rap — 2 (< m — 2). Let np be the outer unit normal at P and 
vp a unit tangent vector at P. Then Cq(A) < oo, if A satisfies, for some e > 0, the Diophantine 
condition 

(1.6) maxsupllfcl^^ 5 " 6 !^,^*^)! : dist(fc,Rn P ) < 1} > 0. 

In particular the set {A € SO(2) : Cn(A) = oo} is of Hausdorff dimension < ^rf • 

It is likely that one can weaken the Diophantine condition and thus the estimate for the upper 
bound of the Hausdorff dimension is presumably not sharp. The latter theorem is related to the 
results by Colin de Verdiere [5] and Tarnopolska- Weiss [30] who proved similar statements about 
the maximal function A i— » sup t>1 t 4 ^ 3 AAn(t)- Also, Nowak [24] has obtained some improved van 
der Corput type bounds |A,4n(t)| < CU£~ 4 / 3 ~ 5 for suitable 5 = 8(d) > 0, again under appropriate 
Diophantine conditions on the rotation. 

We remark that in Theorem 1.2.1 the smoothness assumption can be relaxed considerably; 
moreover a slightly weaker variant of Theorem 1.2.2 holds without any assumption on the boundary 
of the convex domain. These issues are taken up in the sequel [13] to this paper. 

Notation: Given two quantities A, B we write A < B if there is an absolute positive constant, 
depending only on the specific domain O, so that A < CB. We write A w B if A < B and B < A. 



2. Preliminaries 

We denote by fT the polar set of SI, 
(2.1) fT = {£ : (x, £) < 1 for all x e 0}, 

and let p* be the Minkowski functional associated to fT; i.e. p* is homogeneous of degree 1 and 
satisfies p*(£) = 1 if £ € dil,* . Then, if P+(£) is the unique point in <9S1 at which £ is an outer normal 
to dil, then 

(2-2.1) p*(0 = (P + (0,0- 



Similary, if P_ (£) is the unique point in dfl at which £ is an inner normal, then 
(2-2.2) p*(-0 = -(P-(0,0- 



If 1 1 — > x(t) is a regular C k parametrization of <9S1 near a point Po = x(to), and t n(t) denotes 
the outward unit normal vector, then t x*(t) = (x(t), n(i)) _1 n(t) parametrizes the boundary of 
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CI*, and x* is of class C*^ 1 . If k(Pq) denotes the Gaussian curvature at Pq, and k(Pq) ^ then 
the parametrization t x*(t) is regular near P * = x*(t ) and the curvature k*(Pq) of at Pg 
satisfies 

(2-3) \*(PSMPo)\ = (\Po\ ■ \Po\r 4 - 1 - 

For these facts see e.g. Lemma 1 in [21]. 

We shall also need asymptotics for the indicator function of a convex domains. Suppose that 
CI is of finite line type (in the sense that every tangent line has finite order of contact with dCl). 
Let dfjL be a smooth density on the boundary of CI. We define the Fourier transform by /(£) = 
/ f(y)exp(i(—y,£))dy, and then a result by Bruna, Nagel and Wainger [4] says that 

(2.4.1) MZ) = e- 4 < p +^a+(£) + e- l < p -^>a_(£), 

where a± is smooth and satisfies the symbol estimates 

(2-4.2) |^o±(0| < c a7 ±(oi£r H , ICI > 1 

for all multiindices a, and j± is defined as follows. Let Hp be the (affine) tangent plane to CI at P. 
Then 7±(£) is the surface measure of the cap 

(2-5) 7 ±(0 - <7({yedCl:dist(y, H P±{() ) < l^" 1 }), 

where cr denotes surface measure on dCl. By the divergence theorem, d Xi xn = —riidcr, in the 
sense of distributions, where n — (m, . . . , n<j) is the outward unit normal. Thus we get Xn(£) = 
—iJ2i=i(£i/\£,\ 2 ) n id<j(t;). If one combines this with (2.2.1/2) and (2.4.1/2), one obtains 

(2.7) xT 2 (C) = e-*"'(«6 + (0 + e*"*^b-(0, 

where 

(2-8) |^6±(0l<Ca7±(0ier 1_H . ICI>1- 

In the case of non-vanishing curvature one has 7±(£) ^ |^|^( d_1 )/ 2 but of course the above 
statement, and more precise asymptotics, follow from the method of stationary phase as in papers 
by Hlawka [8] (see also §7 in [9]). More generally, for finite type domains one has 

(2.9) 7 ±(o < K*±(0)r 1/2 ir (d - 1)/2 - 

This is proved in [29], and can also be deduced from the cap estimates (2.5) using an argument 
in [6]. However, it should be noted that these results are much easier in the two-dimensional case 
needed here. See [27] and also [1]. 

Definitions. Let S > be fixed so that the ball B 2 s {0) with center and radius 25o is contained 
in CI. Let £ be a smooth nonnegative radial cutoff function supported in the ball 5,5(0) so that 
/ ({x)dx = 1. Let ( E (x) = £- d ((x/e). 

We set N(t) =Af Q (t), 

E(t) = N(t) -t d vo\(Cl), 

and 

(2.10) fcez d 

E e (t) = N £ (t)-t d vol(Cl). 
We also denote by N*(t) and E*(t) the corresponding expressions for the polar domain Ct* . 



Three elementary Lemmas. 

Lemma 2.1. Suppose that has C 1 boundary. Then there is a constant C — C(Q) such that for 
1 < R < t < 2R, < e < 1, 

(2.11.1) \E £ (t - e)| - Ct d ~ x e < \E(t)\ < \E £ (t + e)| + Ct A ~ x e 

(2.11.2) \E(t - e)| - Ct d ~ x e <\E e (t)\ < \E(t + e)\ + Ct d ~ x e 

Proof. By the properties of the the cutoff £ e we have 

N e (t-e) <N(t) < N e (t + e), 

and if we subtract V(t) = t d vol(0) throughout, we get 

E e (t -e) + [V(t - e) - V{t)\ < E(t) < E e (t + e) + [V(t + e) - V(t)]. 

Clearly \V(t ± e) - V(t)\ < t d ~ x e and (2.11.1) follows. (2.11.2) follows as well if we apply (2.11.1) 
with t ± e in place of t. □ 

Lemma 2.2. Suppose that fj, e [0, 1] and that the estimate 

(2.12) supt-( <, - 1 -">|£;(t)| < Ci 

t>o 

holds. Then for t> I 

(2.13) E e (t) <max{t' i - 1 -' 1 ,t d - 1 e}. 
Moreover there a constant C so that for < e < h < r 

/I / >r+/l \i/2 /1 / >r + /l \ 1/2 , 

J \E(t)\ 2 dt) \E e {t)\ 2 dt) I <Cr <, - 1 [ft- 1/ V /2 r-"+e]. 

Proof. We first observe that (2.13) is immediate by Lemma 2.1. We integrate and obtain 

rT+h f-r+h+e 



r-r-yn i>r-\-n-\-E 

/ \E(t)\ 2 dt< / \E £ (t)\ 2 dt + Chr 2d ~ 2 e 2 

J r J r 

rr+h 

< / \E E (t)\ 2 dt + Chr 2d - 2 e 2 + C'er 2< - d - 1 -»\ 

J r 

which implies one of the desired inequalities, the other is obtained in the same way. □ 
Lemma 2.3. Let < e < 1 and let for r > 1 

(2.15) 6(r, e) = card{^ e Z d : r - e < p{t) < r + e}. 

TTien 

r+e/2 



(2.16) e(T,e)<C 1 T d - 1 e + C 2 ( T £ E,{t) 2 N' e {t)di)^ , 

X Jr-e/2 ' 



_ 4e 

/2 #0 



Proof. Let t G (t — e, t + e). We use the elementary inequality 

J X(t+h)n\tn{x - y)e- d ((e~ 1 y)dy > c h/e if /i < e, x <= (r + e)fi \ (r - 
This implies 

7V e (i + JV £ (t) = 2 / X(t+fc)n\tn(* - vK^'^dy 
fc ^ 
ft, 

> c -6(r,e) 

and thus 

(2.17) N' e (t)>c Q e(T,£)£-\ |t-r|<e, e=^. 

We now turn to the proof of (2.16). Wc may assume that S(r, s) > C\ET d ~ x with C\ = 
d2 d+1 c^ 1 vol{Q). Then by (2.17), 

E' e (t) = Nft) - di*- 1 vol(fi) 

> iV e '(*) -d(2T) d - 1 vol(fi) 

> c 6(r, e) £ - 1 -2 d Ci 1 de- 1 vo\(n)6(T,s) 

(2.18) > c (2£)- 1 6(T,e). 

Let J Ti£ = [t - e/2, r + e/2] and pick t G 7 T)£ so that min te / T s |-E e (t)| = |-E £ (<o)|; thus |i? e (t)| > 
\E € (t) - E £ (t )\/2 and |£ e (i)| > \ J^E' £ (s)ds\/2 > c (4e)- 1 |t - t |6(r, e). We use also (2.17) and 
obtain that 



/ ^ ^(t) 2 iV e '(t)dt > / ^ (||6(r, £ )) 2 |i-t | 2 f e(r,£)dt>c[e(T,£)Y 

Jr-e/2 Jr-e/2 

as asserted. □ 



3. Proof of Theorem 1.1 

In this section we assume that has a smooth boundary with everywhere non-vanishing cur- 
vature. This implies that fi* is also smooth and has everywhere non- vanishing Gaussian curvature. 
See (2.3) above. We estimate the square-function 



/ 1 f 2R \ 

gar) = (rJ r i^wi *) 

for < e < 1/2 and R > 2, and set 



1/2 



(3.1) w d (R) = { 



( R?- d if d > 4 

ORlogi?)- 1 if d = 3 

R- 1 ' 2 if d = 2 
6 



and for < s < 1/2 let 

(3.2) M»)= SU P SU P(! +eR)~ d ~ 1 w d (R)G s {R). 

s<e<l/2 R>2 

Analogously, we denote by A d (s) the corresponding quantity associated to 0*. It is not hard to see 
that Ad(s) is finite for every s since we have a trivial estimate Ad(s) < svp R>2 (l + sR)~ d ~ 1 R < s _1 , 
and, similarly, A d (s) < s" 1 for every s < 1/2. We shall see that Ad(s) is bounded as s — > 0. Once 
this is established, the required bound for Qq follows from 

(3.3) gn(R)<R- d (G 1/R (R) + R d - 2 ), 

which is a consequence of Lemma 2.2. 

The boundedness of A d (s) can be deduced from the following iterative procedure. 

Proposition 3.1. There is a constant Cq so that for s < 1/2 

(3.4) A d (s) 2 <Cn(l + A d ( s )). 

Indeed, since 0** = O, (3.4) implies that A* d (s) 2 < Cq* (l + A d (s)), so 



A d (s) 2 < C n (l + A/Cn.(l + A d (a))) 
from which the boundedness of Ad is immediate. 
Proof of Proposition 3.1. We estimate G e (R) assuming first that 

R~ X < e < 1/2. 

We apply the Poisson summation formula X^ez d /W = {^ n ) d Ylkei d fi^k) to / = xsi(£-) * Cs- 
This yields 

(3.5.1) £ £ (£) = ^(2^) d xS(2^fc)C(27refc). 

We split £ £ (£) = J2± E t(t) b y usin s ( 2 - 7 / 8 ); here 

£+(£) = ^(27rt) d 6 + (27rtfc)cxp(-27rip*(fc)) 

kjtO 

(3.5.2) 

£7(£) = ^(27rt) d 6_(27rtfc)cxp(27rip*(-fc)). 

k=£0 

Now fix a nonnegative n £ C°°(R) so that 77(f) = 1 for £ £ [1, 2] and 77 is supported in (1/2, 3). 
Then 

G £ GR)<G+(i?)+G £ -CR) 

1/2 



We shall only consider estimates for Gf(R) because the estimates for G e (R) are exactly analogous. 
Multiplying out the squared expression we get 

(3.6) Gf(R) 2 = C(27refc)C(2TCfc')-R _1 / e 2 * lt{p * {k) - p * {k,)) q Kk ,(t)dt 
where 

(3.7) q k ,k'{t) = b+ (2ntk)b + {2M)t 2d r 1 {t/ R) . 

Thus q kik > is supported in [R/2,3R] and by (2.8) and 7±(£) = _(d_1)/2 ) we have the symbol 
estimates 

(3.8) |(|) m gfefc ,(t)| < CmR d-l-rn lkr V + l)/2 lk , r «+l)/2_ 

We now integrate by parts in t. We note that |fc| w p*(k) and \((2nk/R)\ < C N (1 + \k/R\)~ N and 
obtain the estimate 

Gt(R) 2 < C m ,n £ E + - /°*( fc ')l)- M (l + e\k\ + e\k>\)- N \p'{k)p'(h>)]-*P. 

The terms with |p*(fc) - p*(k')\ > R- 1 ' 2 give a contribution of 0(R d - 1 - M / 2 e- 2d ) = 

Q( R 3d-l-M/2j and we may cnoose M = Q d 

Thus 

-_Rl/2<„<Hl/2 fc^O |p*(£)-p*(fc)P ' 

(3.9) < C^ 1 J] (! + ")~ M EC 1 + ^(fc))^ 6 * ( ^ + y } + CSi**- 1 -"/ 8 ; 

-_R 1 /2<„<i ? l/2 fc/0 ^ ^ ' 



here recall that 6*(r, e) = card{^ e Z d : r - e < p*(£) < r + e}. Now 

Biw<*>r»5W£> 

<E2-'(l+ £ 2')- JV (^ E [6*(p*(fc),^)] 2 ) 



J=0 2 ? <p*(fc)<2 i + 1 

oo 

< ^ 2-'(l + e2 l )- N [(n + 1)J, + //„,,] 



where 



*i = Qs E P^r^R-f 2 , 

2 ; <p*(fe)<2 ! + 1 



_ /J_ ^ ^«(fc)+(n+i)/2a E *{t) 2 N*'(t) 



with e = 4e/<5 ; here we used Lemma 2.3. Observe that for N large, 
(3.10) 

oo oo ( p_i 2-d :f J q 

I «" 1 log(2 + £- 1 ) ifd = 2 

and thus, since we are assuming e < 

oc- 

jjd-i ^ 2 -i^ + e2 ')-^ /; < max { e 2 - d , log(2 + e- 1 )} < w d (R)~ 2 . 

1=0 

We now estimate II n> i and set J fcj „ := [p*(k) — (n + 1)/2R, p*(k) + (n + 1)/2R]. Observe that 

<5(p*(k), ^) - card{£ : p*(k) - ^ < p*{l) < p*{k) + =±1} 

> card{^ : t - ^ < p*(£) < t + ^} if \t - p*(k)\ < 

which is saying that &{p*(k), > 6{t, ^) if t € J fc ,„. Thus 
Therefore 



TI*-2-uf\ V v ml E W) 2 W At 

J 2'<p*(fc)<2' + i ^ ^ W> R > 

< ^(2 d -6+^) + 2 i(2d- 3 ) e 3 + E ;(t) 2 dt) ; 



2 (ft 



here we have used the estimate \E*(t)\ < 2 l(d ~ 2+ d+i ] + 2 li - d -^e 1 t w 2 Z , which by Lemma 2.1 is a 
consequence of the classical estimate \E*(t)\ = 0(t d ~ 2+ ~), d>2. Thus 

oo 

R d-iJ22- l (l + e2 l )- N II n ,t 

1=0 

oo o 2^ 

<R d - 1 (Y,( 1 + £2l y N [ 2l{d ^ + ~ l) + 2^- 5 / 2 )e 3 / 2 + 2-'(l / |££(i)| 

oo 1 

< i? d-i £ 3-<i + i?( i-i^ 2 -;( 1 + e2 -i)-Ar ^ G e (2 1 - 1 ) 

1=0 *=-l 

(3.11) 

< ^d-i^-d + ^-i 2 -/ (1 + £2 -iyN [ 1 gup | (1 + e2 '')- d - 1 G:(2'') U ; d (2 r )}. 

^ w d (2') r > 



Now since R d 1 e 3 d < w d (R) 2 for e > R 1 we have 

R d-i J- 2- l {w d (2 l )Y\l + e2- l y N (l + e2- l ) d+1 
1=0 

oo 

<R d - 1 Y / 2- l (w d (2 l ))- 1 (l + e2- l y 



-N+d+l 



(3.12) 



z=o 

-2 



<w d {R)-\l + eR) d -\ 



where the third inequality follows in a straightforward manner from the definition of w d . It is 
precisely at this point where one needs to distinguish the cases d = 2, d = 3 and d > 4. Combining 
the previous estimates (3.10), (3.11) with (3.12) we obtain for s < 1 and max{s, R^ 1 } < £ < 1/2 

[(l + sRy^WdWG+iR)] 2 

<l + (l + eR)- 2d - 2 w d (R) 2 R d - 1 J2a + ^ l r N E (l + n)" 3 ((n + lKi+^n,i) 



|n|<fli/2 



(3.13) 



< 1 + sup {(1 + e2 r )- d - 1 G:(2 r )w d (2 r )} 

r>0 



for e = 4e/<5 The same estimate holds for G e (R) and thus for G e (R). Consequently, since ewe, 
we have 



(3.14) 



[(1 + eRy^WdWGeiR)] 2 < C(l + A* d {s)) if iT 1 < e < 1/2 



The required estimate for e < 1/R follows from a small modification. Namely we can use Lemma 
2.2 to see that 



r / 1 f 2R \ !/ 2 

G *( R ) ^ C i[(r J \E{t)\dt) +R 



R 

/ 1 f 2R \ 1/2 



Thus 



(3.15) 



(l + ei?)- 2 < d+1 ^ d (i?) 2 G £ (i?) 2 

^(^[iJ^ + G^OR) 2 ] 

<C(l + ^(s)) if s^e^iT 1 . 



The desired estimate (3.4) follows from (3.14), (3.15). □ 



4. Localized square functions in the plane 



In this section we give the simple proof of Theorem 1.2.1. We assume that f2 is a convex domain 
in the plane, with smooth boundary, and that the curvature does not vanish at the boundary. 
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We may apply Lemma 2.2 with p = 0, say and we let 1 < h < R and e = R . Then 
/I r R+h \i/2 r/1 / >K+/l \ 1/2 ,n 

(4.i) (_y i^(t)i 2 dt) <[( x y i^i/fl(*)i 2 *) +{R/h) 1 ' 2 . 

Let 770 be a nonnegative C°° function supported in (—1/2,3/2) and which equals 1 on [0, 1]. Then 
(4-2) - J r \E 1/R {t)\Ht <J2hJ \ E t/R^)W-^)dt 

with as in (3.5.2). The expressions on the right hand side are estimated by integration by parts, 
as in the previous section. We square the series. The cutoff rj(t/R) is now replaced by %(^r^) an( l 
this affects the argument since in the symbol estimates for the modification of q^^i the estimate 
R d-i-m in (- 3 8 ) ig now re pi ace d by R^h,-" 1 . 

As a result we obtain the estimate 
I J \E± R (t)f m (^)dt 

<rEE( 1 + h \p*w p*i k ')\r M i l + \ k \i R + \k'\/R)- N \p*{k)p*{k')\-^ 

and this term is estimated by a constant times 

( 4 - 3 ) E E (! + wr V w _3 (i + pwar^evw + + i? 1 ^ 72 , 

|n|<ijV2 fc^o 

where, as before, 6* (r, e) = card({^ eZ 2 : r - e < p* (£) < r + e}) . 

Now by the classical estimate for the remainder term E{t) with t = p*(k) + (n ± l)/h m p*(k) 
we have 

(4.4) &*( P *(k) + 1 , i) < rV'W + P*(fc) 2/3 . 

Putting the previous estimates together, we have 

E^ /R (t)\ 2 V0 (i^)dt < i?E(! + R ^P*( k )r N ^{h- l p*(k)-\ p*(fc)- 7 /3 } + i?l-M/2 
fc^O 

< i?(l + /i" 1 logi?) 
which is O(-R) if h > logi?. This finishes the proof of Theorem 1.2.1. □ 

5. Estimates for finite type domains in the plane 

We shall give a proof of Theorem 1.2.2. Let 11 be a convex finite type domain in R 2 which 
contains the origin in its interior. We first give a version of the standard lattice rest estimate for the 
polar set ft* which has a C 1 boundary. 

Lemma 5.1. We have the following estimate for the Fourier transform of the characteristic function 
of SI*, 

(5.1) \x^(o\<c(i + \t\r 3/2 . 



Taken Lemma 5.1 for granted we obtain as a consequence 
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Corollary 5.2. Let be a convex set in R 2 , containing the origin in its interior and suppose that 
has smooth finite type boundary. Let O* be the polar set. Then 



(5.2) J\fn* (t) = t 2 area(n*) + 0(t 2 / 3 ) 
as t — ► oo . 

Proof. This follows from Lemma 5.1 using the standard argument (see e.g. [8], or §7 of [9]). 

The Corollary can be improved by using more sophisticated techniques which however are not 
needed here. 

Before proving Lemma 5.1 we recall some terminology: We denote by T the set of all points in 
90 at which the curvature vanishes; these points are separated and thus T is finite. For every PeT 
let mp be the type at P (i.e. the curvature vanishes of order mp — 2 at P). For every P € 90 there 
is a unique P* G 90* so that (P, P*) = 1 and we define T* = {P* : P e T}. 

Proof of Lemma 5.1. 

The boundary 90* is smooth away from T* and it is C 1 everywhere. Thus surface measure da is 
well defined and by an application of the divergence theorem as in §2 estimate (5.1) follows provided 
we can show that 

(5.3) 1^(01 <(i + ieir 1/2 

for X e qf. 

To see this we introduce a partition of unity xdc = J2v Xvdo where each P* e T* lies in exactly 
one of the supports of the functions Xv Clearly it suffices to prove the estimate du u (^) — 0(|CI _1 ^ 2 ) 
for each o v := Xvda. 

Fix v and P G V. If P* ^ supp do~ v then da v {^) = 0(|^| -1 / 2 ) by the standard stationary phase 
argument. Thus suppose PeTn supp da v . By a rotation we may assume that np — (0, 1) and by 
an additional translation we may also assume that P lies on the X2-axis. Let m = mp be the type 
at P. Near P the boundary of O is parametrized by (t, f(t)) where 

4-m 

f(t)=a -a m — + t m+1 gi (t) 
m 

with a > 0, a m > 0. Thus a parametrization of 90* near P* = (a^ 1 , 1) is given by 

n(t) 1 (-/'(*),!) 

(x(t),n(t)) f(t)-tf'(t) + 

however this parametrization is not regular. Denote by ui(t) the first coordinate of (x(t),n(t))~ 1 n(t). 
Then it is easy to see that 

w(t) = (a m /a )t m - 1 (l + t 92 (t)) = Mi))™- 1 
where c = (a m /a ) 1/(m_1) and s(t) = t + 0(t 2 ). Moreover 

(f(t) - tf'{t)y\^iT7W)- 1 = ^(i - ^r^ m + t m+1 92(t)). 
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Thus setting r = (c s(t))" 1 1 we see after a short computation that near P* the boundary is 
parametrized by r i— > (t, h{r)) with 

h(r) = a^(l - d^/f™" 1 ) + r^ ff3 (r^T)) 

where ci = (m — l)m~ 1 (a m /a a )cQ m = (m — l)m _1 (a m /ao) _1 /( m_1 ) and 53 is smooth. Thus we 
have to show that 

(5.4) j(o = J e -^ iT+£2,i(r)) ?7(T)dT = o(ier 1/2 ) 

as |£| — > 00; here we may assume that the support of 771, is contained in a small interval (—5,5). 

It suffices to estimate the analogous integral extended over the set {r : |r| > ICI^ 1 ^ 2 since the 
error is 0(\^ 1/2 )- Observe that for small r we have \h'(r)\ < 1 and \h"(r)\ > CT -(™-2)/(m-i) > L 
Thus by van der Corput's lemma ([28], ch. VIII. 1) we obtain for large |£| the estimate | J(£)| ^ 
if > IC2I (using first derivatives of the phase function) and the estimate |J(£)| < l^l -1 ^ 2 if 
IC2I > (using second derivatives). This implies (5.4) and thus (5.3). □ 

Proof of Theorem 1.2.2. We shall decompose the Fourier transform of xn as in [27], following 
rather closely [13]. Using the divergence theorem as above, we see that 

(5-5) ^(0 = iJ2 |§2 J^(y)e- t{y ^da(y) 

where rii denotes the i th component of rip. 

For every PeTwe choose a narrow conic symmetric neighborhood Vp of the normals {±np}, 
a small neighborhood Up of P in S and a C§° function xp whose restriction to £ vanishes off U and 
so that xp equals one in a neighborhood of P. We may arrange these neighborhoods so that the sets 
Vp H {£ : |£| > 1}, P € r are pairwise disjoint and that the normals to all points in a neighborhood 
of Up are contained in Vp, so that the Up's are disjoint too. 

Define 

Fi, P (0 = [ Xp(y)n l (y)e-^da(y) 

Let »p a unit tangent vector to dfl at P. Then if the cones Vp are chosen sufficiently narrow, we 
have 



(5.5) 

where 
(5.6) 

with 
(5.7) 



ICI-^Hminlier^^-'ep^)} 



-N 



if £ e V> 



6p(0 
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" 2(mp -1) 



This follows from (2.8) (with a = 0) and (2.9) by a straightforward computation. Moreover 

(5-8) £ Y^FiAO) =e-' , <«c + (0 + e*'*<-« C _(0 

i=i 1^1 per 

where 

(5.9) |^c±(0|<C a ^|- 3 / 2 -H. 

The estimate for £ G Vp follows from Proposition 1.2, and the estimate for £ ^ Vp follows by 
a simple integration by parts; namely if t i— > 7(f) parametrizes E near P then |(7'(t),£}| ~ |£| for 
7(i) G J7 P and £ ^ Vp. 

Moreover by the usual stationary phase or van der Corput estimate we have 

(5.10) \m)-Y F ^A0\<^ + \^\r 1/2 

Per 

here we used the definition of T and the fact that %p is equal to 1 near P. 

Let E 1 / R A {t) be the remainder term (2.10) with e = 1/R, with f2 replaced by the rotated 
domain AQ ; that is 

Ei/ R , A (t) =J2xm* Ci/niA^k) - t 2 area(ft) 

feez 2 

(5.11) = Y, t^R-'Ak) XrtAW F ^ tAk ) 

k^O i=l ' ' 

For P G r, A G 50(2) let 

Zf(A) = {fc G 7L d : Ak G V P ,k ^ 0,dist(Afe, M.n P ) < 1} 
Zfj(A) = {keZ d :AkeV P ,k^ 0, dist(Afc, Rn P ) > 1} 

and let 

Z IH {A) = {k G Z d : k ^ 0, fc i Uper^p}. 

We may use estimate (4.1) which does not depend on any curvature assumptions and see that it 
suffices to estimate the square function (h^ 1 J \E 1 / R A (t)\ 2 r]o(^^)dt) 1 ^ 2 (cf. (4.2)). We decompose 
for R < t < 2R 



jw*)=(E E +E E ^R-'Ak)± 2 -^^F lA ^tAk) 

PeTkezf(A) PeTkezfjiA) »=i 1 1 

+ E t^R-'Ak) E Xltlkf ( F ^ tA V E FiA^tAk)) 
keZui(A) »=i 1 1 Per 

+ EE H{2,R^Ak)Y^^F^tAk) 



=E ( E J p(*) + E iitw+iii*®) +iv(t) 



± Per Per 
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where 



(5.12) \IV(t)\=0(t- N ) 
and 

I${t,A)= J2 C(27ri?- 1 Afc)6 + (2 7 rtAfc)e- 27r4 *"*^ 
kezf(A) 

II+(t,A)= J2 C(^R' 1 Ak)b + (27rtAk)e- 27rUp * {Ak) 
kezf^A) 

III+(t,A)= C(^R' 1 Ak)c + (2TrtAk)e- 2mtp ^ Ak \ 
kezf u (A) 

and the expressions Ip, lip and I Hp are defined by replacing b + by c + by c_, and e ~ 2mtp (Ak ^ 

by e 2TTitp*(-Ak)_ 

The argument in the previous section applies to the square functions associated to I// ± (t, A) 
and we obtain the bound 

(5.13) i y |m ± (M)| 2 7 7o (Sr)*^( 1 + ^ llo g i? )> 

uniformly in A. 

A small variation of this argument also applies to the square function associated to IIp(t,A). 
Namely, arguing as in §3 and using (5.6/7) we see that 



I J \II+(t,A)\ 2 m (^)dt 



<9 Wl ,M n . M w *<Ah>\\\-Nh > l fc l + l fc, K -^ ®p{Ak)Q P (Ak') 

- 2 \ R(l + h\ P (Ak)- P (Ak)\) (1 + — _) p * {Ak)3/2p * {Ak , )3/2 



kezf r (A) 
k'ezftiA) 
e P (Ak)>e P (Ak') 



E %f E (i+nr- ( i + | fc |/i?r-6^(^) + f,i)+i? 4 -- 

|n|<2* 



where now 6^(r,e) = card({^ e Z 2 : r - e < p*(A^) < r + e}). 

Observe that dist(Afc, Rn P ) > 1 and dist(Af, Ah) < 1/2 implies that <d P (Ak) w 6 P (A£). Thus 
we can use the argument in §3 and Lemma 5.1 and estimate 

1 rR+h 

- J r \n±{t,A)\ 2 m {^)dt 

<r[ (i + |p*(e)|)- 3 e 2 P (0(i + fl- V(0) -JV mini/.- V*(0" 2 , p*(fc)" 7/3 R + ^" M/2 - 



Since Qp is homogeneous of degree and integrable over the sphere {p*(r]) = 1} it is easy to see 
that the former expression is bounded by R(l + h^ 1 log J?), thus 

i r-R+h 

(5.14) -/ \II+(t,A)\ 2 dt< Ril + h- 1 log R), 

n Jr 

for \h\ < R, uniformly in A. The same estimate holds true with Hp replaced by Hp- the proof only 
requires changes in the notation. 

In order to estimate the square function involving Ip~ , we let Sp(A) be the set of all k € Z 2 \ {0} 
with dist(k,RA*n P ) < 1, and define 

M Pi£ (A) = sup{|fcr 1+£ e P (fc) : k e S P (A)}. 



Then 



I J \iMt)\ 2 voC-i?)dt 



< e e + h\ P * { Ak) - pW)d-"(i + j^)-^ e ^ff y ) 

fceS P (A)fc'eSp(A) 11 11 

<Mp £ (A) 2 ^ i?(l + ^|p*( J 4fc)- /9 *(Afc')|)" Ar (l + ^-^)" Ar |fcr^ 1/2 |fc'r £_1/2 
fceSp(.4) 

<m p , £ (a) 2 ^ jza+ifci/^-^ifcr 1 - 26 , 

fc€S P (A) 

and thus 

(5.15) i | \I+(t)\ 2 Vo (i^)dt < C £ Af P , £ (A) 2 i?. 

Again the same estimate remains true for Ip (t) . 

For each k ^ the function A 9 P (Afc) belongs to the space z,(2mp-2)/(m P -2),oo_ F or a > 
the set {A e 5*0(2) : M Pj£ (4) > a} is the union of the sets £ fe (a) = {A : e P (Ak) > |fc| 1_e a}, 
k e Z 2 \{0} and the measure of £ fe (a) is < (fci-£ Q ,)-(2» l p-2)/(m p -2)^ gincc (2m P - 2)/(m p - 2) > 2 
we may sum over all fc € 1? \ {0} and we see that M Pt£ e L( 2m * , - 2 )/(™p-2),°°( 1 S'0(2)) provided that 
e < 1/2. Combining the estimates (5.12-5.15) this proves that C n € L^p- 2 )/^- 2 )/*^^)). 

The Diophantine condition (1.6) for some e > is equivalent with the condition Mp tS (A) < oo, 
for some £ > 0. Fix P. The estimates (5.12-15) show that Cn(A) = oo also implies Mp_ e (A) = oo for 
at least one P e T. Thus we can complete the proof if for any sufficiently small e > we demonstrate 
that the set {A £ 50(2) : M P , e (A) = oo} has Hausdorff dimension < (m P - 2){m P - 1) _1 (1 - e) -1 . 

Set (3 = (rap — 2)/(2mp — 2), thus (3 < 1/2. Now Mp iC (A) = oo implies that there are infinitely 
many k € Sp(A) so that |fc| £_1 |(fc/|fc|, vp)\~@ > 1. If A*vp = (ai,a 2 ) this means |fciai + fc 2 a 2 | < 
| fc |(/3-i+ £ )//3^ Now | ai | > | a2 | implies < | fc2 | and |q, 2 | > | ai | i m pii es |fc 2 | < j^l (as k e 5 P (A)). 

Thus if |«i| > |ct 2 1 the condition Mp £ (A) = oo implies that for infinitely many k with |fc 2 | w |fc| we 
have that 



(5.16.1) |fci/fc 2 -a 2 /ai| < 0|fc 2 |( £ - 1 )/ /3 or Ifc/fc + a 2 /c*i| < 0|fc 2 | (£ - 1)//3 . 
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Likewise, if \a 2 \ > \oti\ and Mp t£ {A) = oo then 

(5.16.2) \k 2 /k 1 -a 1 /a 2 \<C\k 1 \ {e -W or \k 2 /k x + a x ja 2 \ < C\h \ {£ - 1)/l3 

for infinitely many k with |fci| w |fc|. 

Let Pg denote the set of all x e [— 1, 1] for which there exists infinitely many rationals p/q such 
that \x — p/q\ < q~ 2 ~ e . By a Theorem of Jarm'k [15] (see also [18]) the Hausdorff dimension of Pg 
is equal to 2/(2 + 9) (and we need only the easy upper bound). Now choose in (5.16.1/2) a small 
e > (in particular so that (3 < (1 — e)/2) and we apply the last statement with 9 = (1 — e)^ 1 — 2 
and then 2/(2 + 6) = 2/3(1 - e)" 1 = (m P - 2){m P - l)" 1 ^ - e)" 1 . 

Consequently, with m being the maximal type, the Hausdorff dimension of the set {A e 50(2) : 
Cn(A) = oo} does not exceed (m — 2)/(m - 1). □ 
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